Deterministic Markov Decision Processes Research Articles

Risk probability optimization of finite horizon piecewise deterministic Markov decision processes

This paper investigates the piecewise deterministic Markov decision processes (PDMDPs) under the risk probability criterion. The optimality problem is to minimize the probability that the finite horizon total costs are no more than the cost goal. Under some suitable conditions, the value iteration algorithm is adopted to verify the existence of a solution to the optimality problem. Meanwhile, some new facts are established to prove that the value function is the unique solution to the optimality problem, and the existence of an optimal policy. Finally, two examples are presented to explain the application of risk probability PDMDPs, where the first one illustrates the verification of the main conditions, and the second one shows the calculation of the value function and an optimal risk probability policy.

Stateful Posted Pricing with Vanishing Regret via Dynamic Deterministic Markov Decision Processes

An online problem called dynamic resource allocation with capacity constraints (DRACC) is introduced and studied in the realm of posted price mechanisms. This problem subsumes several applications of stateful pricing, including but not limited to posted prices for online job scheduling and matching over a dynamic bipartite graph. Because existing online learning techniques do not yield vanishing regret for this problem, we develop a novel online learning framework over deterministic Markov decision processes with dynamic state transition and reward functions. Following that, we prove, based on a reduction to the well-studied problem of online learning with switching costs, that if the Markov decision process admits a chasing oracle (i.e., an oracle that simulates any given policy from any initial state with bounded loss), then the online learning problem can be solved with vanishing regret. Our results for the DRACC problem and its applications are then obtained by devising (randomized and deterministic) chasing oracles that exploit the particular structure of these problems. Funding: The work of Y. Emek was supported in part by the Israel Science Foundation [Grant 1016/17]. The work of R. Lavi was partially supported by the Israel Science Foundation [Grant 2560/17] and the National Natural Science Foundation of China [Grant 2560/17]. The work of R. Niazadeh was supported by the University of Chicago Booth School of Business. The work of Y. Shi was partially supported at the Technion by the Council for Higher Education [fellowship], and at Shandong University by the Science Fund Program of Shandong Province for Distinguished Oversea Young Scholars [Grant 2023HWYQ-006].

Compactly Restrictable Metric Policy Optimization Problems

We study policy optimization problems for deterministic Markov decision processes (MDPs) with metric state and action spaces, which we refer to as Metric Policy Optimization Problems (MPOPs). Our goal is to establish theoretical results on the well-posedness of MPOPs that can characterize practically relevant continuous control systems. To do so, we define a special class of MPOPs called Compactly Restrictable MPOPs (CR-MPOPs), which are flexible enough to capture the complex behavior of robotic systems but specific enough to admit solutions using dynamic programming methods such as value iteration. We show how to arrive at CR-MPOPs using forward-invariance. We further show that our theoretical results on CR-MPOPs can be used to characterize feedback linearizable control affine systems.

Planning spatial networks with Monte Carlo tree search

We tackle the problem of goal-directed graph construction: given a starting graph, finding a set of edges whose addition maximally improves a global objective function. This problem emerges in many transportation and infrastructure networks that are of critical importance to society. We identify two significant shortcomings of present reinforcement learning methods: their exclusive focus on topology to the detriment of spatial characteristics (which are known to influence the growth and density of links), as well as the rapid growth in the action spaces and costs of model training. Our formulation as a deterministic Markov decision process allows us to adopt the Monte Carlo tree search framework, an artificial intelligence decision-time planning method. We propose improvements over the standard upper confidence bounds for trees (UCT) algorithm for this family of problems that addresses their single-agent nature, the trade-off between the cost of edges and their contribution to the objective, and an action space linear in the number of nodes. Our approach yields substantial improvements over UCT for increasing the efficiency and attack resilience of synthetic networks and real-world Internet backbone and metro systems, while using a wall clock time budget similar to other search-based algorithms. We also demonstrate that our approach scales to significantly larger networks than previous reinforcement learning methods, since it does not require training a model.

An Accelerated Newton–Dinkelbach Method and Its Application to Two Variables per Inequality Systems

We present an accelerated or “look-ahead” version of the Newton–Dinkelbach method, a well-known technique for solving fractional and parametric optimization problems. This acceleration halves the Bregman divergence between the current iterate and the optimal solution within every two iterations. Using the Bregman divergence as a potential in conjunction with combinatorial arguments, we obtain strongly polynomial algorithms in three applications domains. (i) For linear fractional combinatorial optimization, we show a convergence bound of [Formula: see text] iterations; the previous best bound was [Formula: see text] by Wang, Yang, and Zhang from 2006. (ii) We obtain a strongly polynomial label-correcting algorithm for solving linear feasibility systems with two variables per inequality (2VPI). For a 2VPI system with n variables and m constraints, our algorithm runs in O(mn) iterations. Every iteration takes O(mn) time for general 2VPI systems and [Formula: see text] time for the special case of deterministic Markov decision processes (DMDPs). This extends and strengthens a previous result by Madani from 2002 that showed a weakly polynomial bound for a variant of the Newton–Dinkelbach method for solving DMDPs. (iii) We give a simplified variant of the parametric submodular function minimization result from 2017 by Goemans, Gupta, and Jaillet. Funding: This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme [Grants 757481-ScaleOpt and 805241-QIP].

Risk-sensitive infinite-horizon discounted piecewise deterministic Markov decision processes

Risk-sensitive infinite-horizon discounted piecewise deterministic Markov decision processes

First Passage Risk Probability Minimization for Piecewise Deterministic Markov Decision Processes

First Passage Risk Probability Minimization for Piecewise Deterministic Markov Decision Processes

Neural‐augmented two‐stage Monte Carlo tree search with over‐sampling for protein folding in HP Model

This paper proposes a novel Monte Carlo tree search (MCTS) algorithm to solve the protein folding problem in HP model. There are two main challenges. First, the problem is proved to be NP‐complete. The solution space is large and it is hard to find a good solution via a search algorithm without prior knowledge of the HP model. We tackle this challenge by formulating the problem as a deterministic Markov decision process (MDP) and solve it in an AlphaZero's manner. The difference is that we design a MCTS algorithm with two stages: neural exploitation stage and random exploration stage. In the first stage, the search algorithm utilizes the knowledge from previous experience by evaluating the states with a trained neural network, while in the second stage, the states are evaluated by fast and random rollouts. It effectively reduces the number of neural inferences and computational cost. The second challenge is that the evaluation of typical MCTS cannot preserve the correct preference over the actions in our task. To address this challenge, we propose an over‐sampling mechanism that encourages the agent to search more on those actions with high rollout values. The proposed method is tested and compared in a series of experiments. Experimental results have confirmed the effectiveness of the proposed method empirically. Besides, we also visualize the obtained the best conformations and verify the proposed technical designs through an ablation study. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Adjacency Constraint for Efficient Hierarchical Reinforcement Learning.

Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is large. Searching in a large goal space poses difficulty for both high-level subgoal generation and low-level policy learning. In this article, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a k-step adjacent region of the current state using an adjacency constraint. We theoretically prove that in a deterministic Markov Decision Process (MDP), the proposed adjacency constraint preserves the optimal hierarchical policy, while in a stochastic MDP the adjacency constraint induces a bounded state-value suboptimality determined by the MDP's transition structure. We further show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks including challenging simulated robot locomotion and manipulation tasks show that incorporating the adjacency constraint significantly boosts the performance of state-of-the-art goal-conditioned HRL approaches.

Using Environmental Annotations and Affordances to Model Culture

This paper details the demonstration of an annotation and affordance-based software model intended to introduce cultural and social influences into a non-player character’s (NPC) decision-making process. We describe how recent research has supported the need to begin incorporating the effects of culture into the interactive digital domain. The technical approach is presented that describes the software techniques for embedding and utilizing culturally-specific information inside of a virtual environment, as well as the design and implementation of a deterministic Markov Decision Process (MDP) to model the affects of culture on the AI.

L^*-based learning of Markov decision processes (extended version)

Automata learning techniques automatically generate systemmodels fromtest observations. Typically, these techniques fall into two categories: passive and active. On the one hand, passive learning assumes no interaction with the system under learning and uses a predetermined training set, e.g., system logs. On the other hand, active learning techniques collect training data by actively querying the system under learning, allowing one to steer the discovery ofmeaningful information about the systemunder learning leading to effective learning strategies. A notable example of active learning technique for regular languages is Angluin’s L^*-algorithm. The L^*-algorithm describes the strategy of a student who learns the minimal deterministic finite automaton of an unknown regular language L by asking a succinct number of queries to a teacher who knows L.In this work, we study L^* -based learning of deterministic Markov decision processes, a class of Markov decision processes where an observation following an action uniquely determines a successor state. For this purpose, we first assume an ideal setting with a teacher who provides perfect information to the student. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling execution traces of the system via testing.Experiments performed on an implementation of our sampling-based algorithm suggest that our method achieves better accuracy than state-of-the-art passive learning techniques using the same amount of test obser vations. In contrast to existing learning algorithms which assume a predefined number of states, our algorithm learns the complete model structure including the state space.

An agile and intelligent dynamic economic emission dispatcher based on multi-objective proximal policy optimization

The recent development in power systems claims the improvement of the efficiency to solve dynamic economic emission dispatch (DEED) problem. However, the popular pure optimization framework based on evolutionary algorithms only starts the compute-intensive optimization process after receiving the power demand, which causes significant delays To address this limitation, this research defines a novel dynamic economic emission dispatcher (DEEDer) problem and the dispatcher learning framework. It is different from previous research and the most current algorithms in the advantage of transferring the on-line compute-intensive optimization task to off-line. To solve the dispatcher, we model the dispatching process as a conditional deterministic Markov Decision Process (MDP) and propose the multi-objective proximal policy optimization (MOPPO). The Benchmark Test Set with 10,000 different dispatching tasks for 5-unit and 10-unit system is released to evaluate the generalization of the dispatcher. The experiment results indicate that the neural network dispatcher trained with MOPPO is hundreds to thousands of times faster than the state-of-the-art pure optimization algorithms. Meanwhile, the dispatcher not only has comparable performance as the state-of-the-art multi-objective optimization algorithms in standard dispatching task but also shows generalization on generating Pareto optimal solutions given any possible dispatching task. Following the DEEDer framework, the proposed method makes it possible to dispatch power in a more agile way as the timely response to the changing of power demand while still controlling economic and emission.

A useful technique for piecewise deterministic Markov decision processes

This note presents a technique that is useful for the study of piecewise deterministic Markov decision processes (PDMDPs) with general policies and unbounded transition intensities. This technique produces an auxiliary PDMDP from the original one. The auxiliary PDMDP possesses certain desired properties, which may not be possessed by the original PDMDP. We apply this technique to risk-sensitive PDMDPs with total cost criteria, and comment on its connection with the uniformization technique.

Fuzzy Approximate Learning-Based Sliding Mode Control for Deploying Tethered Space Robot

This article proposes a hybrid control scheme synthesizing fuzzy approximate Q-iteration algorithm and discrete-time terminal-like sliding mode control for deploying tethered space robot, which is modeled as a deterministic Markov decision process. The existence of a switching condition allows FQ-iteration algorithm and terminal-like sliding surface constituting an optimal sliding mode control, and the fuzzy logic approximation is employed to improve the efficiency of optimization. Under arbitrary switching, the sliding mode reaching law works to compress the contraction of sliding surface variable. Simulation results verify the analyses on contraction of fuzzy approximate Q-iteration for optimal sliding mode control, the stability of reduced-order system yielded by the proposed discrete-time terminal-like sliding surface, and existence of switching condition.

Generating Explanations for Temporal Logic Planner Decisions

Although temporal logic has been touted as a fruitful language for specifying interpretable agent objectives, there has been little emphasis on generating explanations for agents with temporal logic objectives. In this paper, we develop an approach to generating explanations for the behavior of agents planning with several temporal logic objectives. We focus on agents operating in deterministic Markov decision processes (MDPs), and specify objectives using linear temporal logic (LTL). Given an agent planning to maximally satisfy some set of LTL objectives (with an associated preference structure) in a deterministic MDP, we introduce an algorithm for constructing explanations answering both factual and “why” queries, which queries are also specified in LTL.

Max-Plus Linear Approximations for Deterministic Continuous-State Markov Decision Processes

We consider deterministic continuous-state Markov decision processes (MDPs). We apply a max-plus linear method to approximate the value function with a specific dictionary of functions that leads to an adequate state-discretization of the MDP. This is more efficient than a direct discretization of the state space, typically intractable in high dimension. We propose a simple strategy to adapt the discretization to a problem instance, thus mitigating the curse of dimensionality. We provide numerical examples showing that the method works well on simple MDPs.

Dynamic Equilibrium with Randomly Arriving Players

There are real strategic situations where nobody knows ex ante how many players there will be in the game at each step. Assuming that entry and exit could be modeled by random processes whose probability laws are common knowledge, we use dynamic programming and piecewise deterministic Markov decision processes to investigate such games. We study these games in discrete and continuous time for both finite and infinite horizon. While existence of dynamic equilibrium in discrete time is proved, our main aim is to develop algorithms. In the general nonlinear case, the equations provided are rather intricate. We develop more explicit algorithms for both discrete and continuous time linear quadratic problems.

Multiconstrained Finite-Horizon Piecewise Deterministic Markov Decision Processes with Unbounded Transition Rates

This paper studies a multiconstrained problem for piecewise deterministic Markov decision processes (PDMDPs) with unbounded cost and transition rates. The goal is to minimize one type of expected finite-horizon cost over history-dependent policies while keeping some other types of expected finite-horizon costs lower than some tolerable bounds. Using the Dynkin formula for the PDMDPs, we obtain an equivalent characterization of occupancy measures and express the expected finite-horizon costs in terms of occupancy measures. Under suitable assumptions, the existence of constrained-optimal policies is shown, the linear programming formulation and its dual program for the constrained problem are derived, and the strong duality between the two programs is established. An example is provided to demonstrate our results.

Scalable Methods for Computing State Similarity in Deterministic Markov Decision Processes

We present new algorithms for computing and approximating bisimulation metrics in Markov Decision Processes (MDPs). Bisimulation metrics are an elegant formalism that capture behavioral equivalence between states and provide strong theoretical guarantees on differences in optimal behaviour. Unfortunately, their computation is expensive and requires a tabular representation of the states, which has thus far rendered them impractical for large problems. In this paper we present a new version of the metric that is tied to a behavior policy in an MDP, along with an analysis of its theoretical properties. We then present two new algorithms for approximating bisimulation metrics in large, deterministic MDPs. The first does so via sampling and is guaranteed to converge to the true metric. The second is a differentiable loss which allows us to learn an approximation even for continuous state MDPs, which prior to this work had not been possible.

Model Predictive Control with a Visuomotor System for Physics-based Character Animation

This article presents a Model Predictive Control framework with a visuomotor system that synthesizes eye and head movements coupled with physics-based full-body motions while placing visual attention on objects of importance in the environment. As the engine of this framework, we propose a visuomotor system based on human visual perception and full-body dynamics with contacts. Relying on partial observations with uncertainty from a simulated visual sensor, an optimal control problem for this system leads to a Partially Observable Markov Decision Process, which is difficult to deal with. We approximate it as a deterministic belief Markov Decision Process for effective control. To obtain a solution for the problem efficiently, we adopt differential dynamic programming, which is a powerful scheme to find a locally optimal control policy for nonlinear system dynamics. Guided by a reference skeletal motion without any a priori gaze information, our system produces realistic eye and head movements together with full-body motions for various tasks such as catching a thrown ball, walking on stepping stones, balancing after being pushed, and avoiding moving obstacles.